Co-producing digital mental health interventions: A systematic review

Objective Smartphone apps (apps) are widely recognised as promising tools for improving access to mental healthcare. However, a key challenge is the development of digital interventions that are acceptable to end users. Co-production with providers and stakeholders is increasingly positioned as the gold standard for improving uptake, engagement, and healthcare outcomes. Nevertheless, clear guidance around the process of co-production is lacking. The objectives of this review were to: (i) present an overview of the methods and approaches to co-production when designing, producing, and evaluating digital mental health interventions; and (ii) explore the barriers and facilitators affecting co-production in this context. Methods A pre-registered (CRD42023414007) systematic review was completed in accordance with The Preferred Reporting Items for Systematic reviews and Meta-Analyses guidelines. Five databases were searched. A co-produced bespoke quality appraisal tool was developed with an expert by experience to assess the quality of the co-production methods and approaches. A narrative synthesis was conducted. Results Twenty-six studies across 24 digital mental health interventions met inclusion criteria. App interventions were rarely co-produced with end users throughout all stages of design, development, and evaluation. Co-producing digital mental health interventions added value by creating culturally sensitive and acceptable interventions. Reported challenges included resource issues exacerbated by the digital nature of the intervention, variability across stakeholder suggestions, and power imbalances between stakeholders and researchers. Conclusions Variation in approaches to co-producing digital mental health interventions is evident, with inconsistencies between stakeholder groups involved, stage of involvement, stakeholders’ roles and methods employed.


Introduction
A global deficit in access to mental health treatment, specifically psychological support 1 has led to the development, evaluation, and/or deployment of digital mental health interventions (DMHIs), including software applications (apps).A DMHI is defined as an intervention delivering information, support, or therapy for mental health conditions through an electronic medium with the aim of treating, alleviating, or managing mental health difficulties. 2DMHIs have demonstrated efficacy, supported by meta-analytic evidence for a range of mental health difficulties, including anxiety 3 and depression. 4,5DMHIs are increasingly regarded as viable alternatives to augment and expand the delivery of mental healthcare. 4,6ith the widespread availability of smartphones, apps are increasingly being used to address the shortage of access to psychological interventions.Although evidence supports the use of apps for the delivery of DMHIs, 7 app use over more lengthy periods of time is not sustained. 8isalignment between researchers' aims and clinical users' needs has been shown to be a major contributing factor to attrition 9 ; specifically, lack of attention to individual user characteristics and poor app design. 10Additionally, apps require users to be intrinsically motivated, as users typically engage with apps in their own time. 11Therefore, key to continued engagement is understanding end users' needs and preferences. 12In this study, the term 'end user' is used to refer to the person who uses or is intended to use the DMHI for mental health support.
Participatory methods provide one means of ensuring that apps are designed to better meet users' needs and therefore promote longer term engagement.The health literature contains various participatory methods that involve different approaches for collaboration between researchers and stakeholders. 13The terminology for these methods is not always consistent, with a multitude of terms used (often interchangeably), including: patient and public involvement (PPI), co-production, co-design and co-creation. 14In the absence of a single agreed term, the term 'co-production' is used throughout this review.Despite inconsistency in terms, a common principle underlines all methods of co-production: 'Nothing about us, without us', 15,p.1 with an underlying ethos of developing with and not for users, 16 by placing end users at the heart of the process.
Co-production has a sound theoretical basis and is grounded in self-determination theory, whereby autonomy and relatedness increase the likelihood of behaviour change. 17Evidence indicates the indispensability of co-production in developing interventions to enhance outcomes.In a comprehensive review of co-production in mental health more generally, commissioned by the mental health charity, Mind, Slay and Stephens 18 examined pertinent themes related to well-being, social connectedness, stigma, and the mitigation of acute mental health service utilisation following co-producing services and interventions.Notably, their findings underscored the impact co-production had on improved functioning, characterised by heightened autonomy, competence and relatednessa congruence with the tenets of self-determination theory. 17he importance of co-produced research is also increasingly recognised by national agendas, including the UK's National Institute for Health and Care Research. 18o-production is deemed to be particularly important as it enables the inclusion of minoritized groups and the accommodation of cultural needs by its ability to address mental health inequalities. 19It is even possible for researchers to prioritise engaging under-representative groups and ensuring that their voices are embedded throughout decision making processes. 20espite the growing recognition of the importance of co-producing DMHIs with people who experience mental health difficulties, 21 few DMHIs have involved co-production processes. 22Additionally, those DMHIs that have been co-produced are often limited to involvement in only the early and/or final stage of research design/intervention development or delivery, 23,24 which is contradictory to the core principles of co-production. 25onsequently, many mental health apps are publicly accessible from app stores and have many monthly active users, 26 but lack sufficient evidence regarding their design, development, and evaluation. 27One possible barrier in involving users in app design is that there is no standardised guidance on how to involve stakeholders in this process.There are several frameworks for involving users in research more generally, but none focus specifically on involving users in the design of DMHIs more specifically.Involving users in the design of DMHIs can bring additional complexity owing to the diverse and multitude of stakeholders required, including researchers, service users (or 'end users'), health professionals and app developers. 28n terms of user involvement in research frameworks more generally, the NIHR INVOLVE produced guidance around co-producing research 25 describing five key principles: (a) the sharing of power; (b) inclusion of diverse perspectives and skills; (c) respect and value the knowledge of those working together; (d) reciprocity; and (e) relationship building and maintenance.Additionally, the UK Design Council's double diamond method advises four main phases where involvement can take place; discover (identify the problem), design (define the intervention), develop (develop potential solutions) and deliver (testing).Another tool is the Involvement Matrix, co-produced by Smits et al., 29 which defines five main roles within co-production (the listener, co-thinker, advisor, partner and decision-maker).The roles reflect the degree of stakeholder involvement, ranging from passive recipient of information (listeners) to active contributors of options (co-thinkers) and advice (advisors), to equal collaborations (partners) and finally, the highest level, the decision makers.In a systematic review of 22 studies, Veldmeijer et al. 30 explored the extent to which stakeholders were involved in mental healthcare through design, including DMHIs.The review found most studies involved end users at a 'co-thinker' level within the matrix framework developed by Smits et al., 29 with no single study involving end users at the highest level as 'decision maker'.These conclusions were supported by another review of 433 studies by Baines et al. 31 who explored co-production within digital health innovation, implementation, and evaluation, and found whilst co-production is recognised as essential, it is rarely practised.Whilst these reviews concern co-production, they did not describe the methods and approaches to facilitate co-production, nor explore the barriers and added value to implementing co-production, which would be clinically useful for guiding decisions when co-producing DMHIs.Furthermore, the studies were not limited to the delivery of DMHIs through apps, despite the widespread availability of apps, 32 and increasing popularity of this platform for mental health treatment. 33ome individual studies have described the way in which they have used co-production methods in the development and delivery of DMHIs.However, to our knowledge, there are no published reviews that systematically aggregate the methods and approaches employed.Therefore, our aim is to systematically review the literature to map out the methods of, and approaches to, co-production when designing, producing, and evaluating apps aimed at supporting individuals with mental health difficulties.Notably, to the best of our knowledge, no previous review has focused on apps, despite their rapid scalability and stakeholders' concern regarding the lack of scientific evidence.We aim to offer an in-depth exploration by synthesising the methods used in co-production of mental health apps and offer recommendations for improving the co-production process to maximise acceptability and engagement with DMHIs.The objectives of this study are to: (i) describe the methods and approaches to co-production that have been used when designing, producing, and evaluating DMHIs delivered via apps; and (ii) explore the barriers and facilitators affecting co-production with DMHIs delivered via apps.

Methods
The systematic review was conducted and reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidance. 34The review protocol was developed in advance and registered with the International Prospective Register of Systematic Reviews of the National Institute for Health Research (PROSPERO CRD42023414007).As this is a mixed studies systematic review, the seven standard systemic review steps for mixed studies reviews have been followed 35 : (i) specifying the review question; (ii) defining eligibility criteria; (iii) applying an extensive search strategy; (iv) identifying potentially relevant studies through rigorous screening (by lead reviewer and an independent reviewer); (v) selecting relevant studies based on full text; (vi) data extraction and study quality appraisal (using the bespoke co-produced quality tool); and (vii) synthesising the data from the included studies.

Search strategy and identification of studies
Relevant studies were identified through systematic search of five electronic databases: MEDLINE, PsycInfo, Embase, CINAHL plus and Web of Science.Databases were searched in June 2023, from inception to present day.A comprehensive search strategy was developed, with a wide variety of key search terms and was performed by primary author (RU).The PICOS framework 35 supported the development of the search strategy.Search terms were also informed by title and abstracts of key papers, including review papers. 30,31Search terms were categorised into: (a) mental health difficulties, (b) mobile applications and (c) co-production.Free text and Medical Subject Headings (MeSH) terms were used for each database.Search terms within each category were combined with Boolean operator 'OR' and the three categories were combined with Boolean operator 'AND'.Pilot searches were undertaken to help generate the final search terms (see Table 1).Reference lists were browsed for any additional relevant studies and citation chasing was undertaken 36 to further supplement the search.

Study selection
Study selection and exclusion processes are outlined in Figure 1.After removing duplicates, article titles and abstracts were screened against the eligibility criteria by author RU.If eligibility was unclear, full text articles were obtained, reviewed in accordance with eligibility criteria and discussed with the wider research team.A second independent rater (SR) screened 15% of titles and abstracts, with any disagreements resolved with the research team.Following this, 15% of the full text articles were also screened by the second rater (SR) to assess reliability of study selection.There was substantial agreement (93%) between raters.Identified discrepancies were resolved by consensus within authors.Disagreements at full text stage were due to whether the app classified as a DMHI targeting mental health difficulties or emotional well-being, and whether there was enough information regarding the co-production process.
Inclusion criteria were: (a) stakeholder involvement (i.e.service users, mental health professionals); (b) studies explicitly describing stakeholder involvement in the design, development and/or evaluation; (c) studies describing a DMHI delivered by a smartphone app designed specifically to target mental health problems; and (d) published in peer review journals, either a qualitative or mixed methods design.DMHIs were defined using the definition from the Hollis et al. (2017, p. 475) review: 'DMHIs as: information, support and therapy for mental health conditions delivered through an electronic medium with the aim of treating, alleviating, or managing symptoms'.Exclusion criteria were : (a) non-human participants or no details around stakeholders; (b) studies with insufficient information around user-centred design; whereby the co-production process if not explicitly outlined, i.e. detailed co-production happened but does not describe involvement; (c) studies not focused on delivering a DMHI aimed at targeting mental health problems (e.g.focused on general wellbeing); and (d) studies not written or translated into the English language, within the grey literature or full texts not available.

Data analysis
Data was extracted using a structured form that was developed and piloted on a small number of papers.Extracted data included: stakeholder characteristics, DMHI characteristics, co-production approach, characteristics (method and outcome of co-production) and reported barriers and facilitators to co-production.Author RU extracted the data from the included papers and a second independent of the research team also extracted the same data from 20% of papers to check for consistency.Any extraction disagreements were resolved within the wider research team.
Due to the absence of an established system for rating co-production in the context of DMHIs, and following the ethos of co-production, a bespoke quality rating method based on the Centre for Reviews and Dissemination guidance for conducting systematic reviews (CRD, 2009) was co-produced with an expert by experience (see Supplementary material).Developing the tool entailed five iterative steps: (a) collaborative consultation to explore the purpose of the tool and discuss roles and expectations; (b) collaboratively reviewing the literature, including amalgamating existing key frameworks (NIHR INVOLVE UK Standards for Public Research 18 ; the 4pi National Involvement Standards 37 ; the UK Design Council's double diamond method (2005); and the Involvement Matrix 29 to synthesis the key phases and determine areas for quality assessing; (c) developing and testing the prototype tool; (d) refining and formalising the tool within the supervisory team; and (e) testing the final tool.Whilst there are several participatory research frameworks, they all report a similar series of sequential phases.The bespoke tool placed emphasis on the steps taken by the researcher to ensure identification and representation of the population, equality in decision making, and the process of involvement.Studies were not excluded based on quality if they addressed the focus of the review.Moreover, the process of quality assessment ensured rich and meaningful insights from the data were not lost.Thus, studies were not assigned a numerical value in the interest of this being an appraisal of each criterion to better describe the quality of the co-production methods of the studies included, and in accordance with Siddaway et al. (2019) who recommends avoiding summary scores.To establish a reliable rating system, 20% of articles were independently rated by an independent second reviewer with areas of discrepancy identified and disagreements resolved through discussion within the wider team.The quality appraisal results have been integrated throughout the results section given the review is focused on appraising the studies.
A narrative synthesis of the evidence was conducted, which involved presenting the characteristics, findings, and quality appraisal alongside the reported barriers and added value of co-production for each study within tables to enable a within-study synthesis.A cross-study synthesis was then conducted whereby the differences between characteristics and quality appraisal ratings of the studies were reviewed and integrated.This supported the ability to aggregate and synthesise the methods and approaches to co-production with mental health interventions delivered via apps, and the barriers and added value to co-production.ENTREQ checklist was used for transparency. 38

Search results
Initial database search yielded 6906 records.After duplicates were removed, 5099 records were screened by title

Study characteristics
0][61] Therefore, these three papers were combined,  totalling 24 studies included for synthesis.Nearly half of the studies (n = 10) were conducted in either UK or USA.Studies targeted a range of mental health conditions: psychosis (n = 8); depression (n = 4); suicidal ideation and crisis support (n = 3); mental health but not otherwise specified (n = 3); anxiety (n = 2); and borderline personality disorder, serious mental illness, relational difficulties and depression and anxiety (1 study each).Not all studies detailed a therapeutic modality; however, 10 studies reported delivering some form of cognitive therapy.Fifteen studies involved adult end users, three studies focused on 18-34-year-olds, three studies involved young people ranging between 12 and 17 years old, one study focused on adolescents aged 12-24-year-olds, and one study did not detail specific ages (but reported mean age = 21.4 years).

Methods and approaches to involvement
The evidence in relation to methods and approaches taken to co-produce DMHIs is summarised below and in further detail in Tables 2 and 3. Overall, only three studies 49,50,59 endorsed all key principles identified in our quality appraisal tool.Furthermore, the identified barriers and facilitators around the co-production process are also described below.

User involvement
Across all studies reviewed, between 5 and 924 stakeholders were involved in the co-production process, with a combined reported sample of 1768.Type of participants and stakeholders included in the co-production process varied, but representatives from the target population ('end users') were included in most studies (n = 21), apart from three. 43,51,52Two studies explicitly chose not to include population representatives due to the perceived sensitivity of the target population (e.g.suicidal ideation 51 ; e.g.refugees). 52For example, O'Grady et al. 52 used agematched controls instead of young people with suicidal thoughts; although, this was recognised as a limitation.Burchet et al. 53 included 128 Syrian refugees, however they were not pre-screened for mental health difficulties, and therefore recognised they did not necessarily represent the DMHIs target group.Additionally, Li et al. 44

Co-production phase
Nineteen of the 24 studies reported co-producing the DMHI from the outset, which aligns with the first stage of 'discover' within the double diamond design framework for developing and evaluating interventions.Twenty-two studies involved stakeholders in designing the DMHI and 17 within the development stage.The final stage of the reported co-production process, prototype testing, was reported in 14 studies.However, an additional three studies did report testing the DMHI, but with 'experts', 41 a design team and students 52 and a sample that may not represent the target population. 53Additionally, less than half of the studies (n = 9) co-produced throughout all four stages of involvement (discover, design, develop and test).

The role of involvement
Whilst all studies made explicit reference to the intended outcomes of stakeholder involvement and utilising co-production methodology, less than half of the studies (n = 10) referred to co-production guidelines.The Involvement Matrix tool 30 to distinguish the five roles of co-production involvement (listener, co-thinker, advisor, partner, and decision-maker) was used to determine the roles of stakeholders in the included studies (see Table 2).In 13 of the studies, stakeholders were classified as having the role of 'co-thinkers', which involves providing opinions throughout each stage.For example, Fortuna et al. 63 conducted a usability test with five end users and developed a second version of the app following user feedback.In 11 studies, stakeholders were classified as 'partners' involved as shared decision makers.For example, in Hardy et al.'s 56 'discover' phase, all stakeholders (service users, carers, therapists and clinicians) developed a shared understanding of psychological therapy, behaviour change, psychosis and technology use from the perspective of various stakeholders, to develop the most intuitive ways of communicating these ideas.The highest form of involvement within the matrix, the 'decision-maker' role, where end users take initiative and/or make final decisions, was not identified in any studies.

Methods of co-production
The methods used to facilitate the co-production process are displayed in Table 4. Methods included more traditional design methodology: interviews (n = 13); focus groups or group discussions (n = 17); surveys or questionnaires (n = 7); consultations (n = 5); reviewing literature (n = 5) and observations (n = 2).More unique methods of co-production involved 'think aloud' or cognitive walk-through methods 65 in eight studies.These methods examine the usability of a product, where stakeholders are asked to carry out tasks while thinking out loud about what they would do and evaluate whether the task at hand is easily achievable.For example, Derks et al. 55 asked end users and therapists to complete tasks whilst asking, "Will the primary user notice that progress is being made toward accomplishment of their goal?"(p. 7).Other methods including creative methods using design artefacts (n = 8), such as storyboarding, card sorting, mock-ups drawings and paper prototypes.App prototype testing, which involves a preliminary visual mock-up that looks like a real app and demonstrates an app's fundamental design and function, was conducted in seventeen studies.App prototypes can be as basic as sketches, or as high-fidelity as a clickable, digital model that works on stakeholders' phones.Prototype testing is believed to be the most effective way of knowing how a product will perform.The final version of apps were tested by stakeholders in laboratory settings (n = 15), or real-world setting (n = 10).Real world testing was achieved through either field testing (n = 3), evaluating the adoption of product features, where stakeholders roam freely by exploring any exhibit they choose to test usability rather than content; or, through beta testing (n = 8), which aims to evaluate satisfaction and ensure release readiness.Beta testing involves a focused tour, where certain functions are directly presented to the user and evaluated.Prototype testing was often combined with other methods.For example, Burchet et al. 53 used 'think aloud' methods based on initial impressions and feedback, while stakeholders engaged with the prototype app, asking, 'Do you think that this app can be helpful for Syrian refugees here in [country] who experience sadness or distress?' (p.5).

Challenges and added value to co-producing DMHIs
Co-production with stakeholders yielded various advantages and obstacles for the development and implementation of DMHIs.

Challenges to co-production
Resource constraints.One of the main challenges encountered in the co-production process was practical constraints.Several studies 43,47,51,53,58,66 reported issues related to resources, time and costs that constrained co-production activities.Researchers recognised resource constraints restricted them from being able to always explore and address all stakeholder suggestions.Time restraints meant that Easton et al. 47 were unable to explore suggestions around creative approaches to engaging people in therapy as they prioritised reviewing app content.
Practical challenges were exacerbated by the digital nature of the intervention development and testing.Some stakeholders did not have access to mobile phones and could therefore not test the tool. 40,46,53In other cases, the expense involved in iterating and creating multiple versions of the app was prohibitive, 51,53,58,66 and technical limitations in incorporating user feedback into app features was also a challenge. 48,50For example, in Berry et al., 66 end users requested multi-media features, such as voice recording features, that were not feasible for the software team to develop due to time and finding constraints, limited resources within the study time frame or the funding of the research grant.Moreover, where researchers asked stakeholders to test a prototype version of the app that did not necessarily reflect the end-product, 53 stakeholders reported feeling less engaged in, and satisfied with, the co-production process.Furthermore, some studies noted that, in some instances, stakeholders changed their views or preferences over time, or hypothetical ideas did not always translate well into practice, resulting in the requirement of further changes and increased costs.For example, two studies reported that initial preferences identified by stakeholders in the 'discovery stage' were not endorsed during subsequent stages (e.g.Chatbot features 44 ; number of alert notifications 66 ).
Recruitment challenges and commitment.A common barrier reported in most studies reviewed 40,47,51,54,55,59,61,62 was the use of small samples to co-produce, reducing generalisability and influencing the nature of the input received.Several studies further noted the lack of recruiting a diverse sample as a challenge, 39,40,[42][43][44][45]51,53 and consequently queried how representative the views were to the wider target audience the DMHI was designed to help. Studis recognised methods used to recruit influenced the limited diversity of the sample, such as recruiting from services, where individuals may be more help-seeking and/or have a better self-management of their mental health.57,58 Researchers also acknowledged online recruitment was likely to target more proficient users of technology, thus leading to a more biased sample in terms of familiarity with, and openness to, smartphones.44,47 Another challenge related to recruitment was around managing risks (i.e.suicidal ideation) within groups of more vulnerable users.51,53,57 Researchers recognised it was important to ensure the safety and well-being of end users in the event they found taking part in co-production work distressing, especially when this involved sharing personal experiences.As indicated previously, the concern regarding managing safety led to one study excluding end users 52 and is at odds of the fundamentals of co-production.26 Furthermore, as Burchet et al. 53 recruited Syrian refugees, audio recordings of the interviews were not made due to privacy concerns raised by the sample.Instead, a written record was made, and researchers recognised their approach to data collection is likely to have affected data quality and depth.

DIGITAL HEALTH
Research teams also had to adapt to the changing circumstances of stakeholders, such as rescheduling meetings due to low attendance. 58,66Berry et al. 66 described how inconsistency with user attendance meant end users were unable to take a leader role to co-chair meetings, and co-production relationships were harder to build.
Conflicting views and expectations.An important aspect of effective co-production was achieving a balance between stakeholders' suggestions, whilst acknowledging the diversity of their needs and preferences.Several studies [43][44][45]47,53,62,66 noted variability in views and opinions expressed about topics covered in co-production sessions, which led to difficult decisions needing to be made by the research team around how to integrate and reconcile suggestions into the next iteration of app development. For exampleBranelley et al. 43 described stakeholders expressing conflicting requests for alert notifications via the app: some stakeholders found alert notifications helpful, while others found these intrusive.Preferences also varied depending on how stakeholders intended to the use the app, with some expressing the need for immediate crisis support as an important feature of the app, and others preferred prioritisation of longer-term features, such as tracking.45 Moreover, two studies 47,66 reported that some user, and even clinician suggestions were at odds with the theoretical underpinnings of the intervention.
Additionally, some studies 44,47,66 encountered challenges in accommodating the preferences across different stakeholder groups.For instance, when Li et al. 44 co-produced a smartphone app for young people with anxiety and depression, parents valued credibility, professionals valued safety, and young people desired accessibility, relatability, and reliance.Other studies identified they were not always able to capture the views of all stakeholder groups, including engaging family members due to recruitment and practical issues, limiting experiences across stakeholder groups. 40wer imbalance.A further barrier included power imbalances between stakeholders and researchers' methodological approaches, including focus groups were described as a contributing factor to an imbalance in power.Some studies felt end users did not voice ideas or talk about personal experiences due to group dynamics. 47,49,53,54,66tudies generally did not report on inclusivity, power, and equality in decision making within co-production processes, which are key principles within the NIHR INVOLVE 26 guidance; with only seven of the 24 studies reporting on any efforts made to ensure the inclusivity and accessibility of co-production activities.Furthermore, ratings on the methodological quality tool (Table 3) demonstrated only thirteen studies reported on attempts to minimise power imbalances.An example of high-quality approaches to inclusivity included Terp et al.'s 39 study who explicitly described how they supported a young adult user with low writing and literacy skills by putting their story into words, and two end users who lacked the confidence to present their ideas to the wider group were supported by team leaders.However, nearly all the studies (n = 22) reported on changes to interventions based on stakeholder feedback, using specific examples.For example, Atif et al. 42 found end users preferred brief messages due to low literacy and effects of depression, therefore the content was broken down into small segments with just one or two key messages, to meet the needs of end users.

Added value
Enhancing cultural sensitivity.The included studies identified the co-production of DMHIs delivered through apps enabled researchers to be culturally sensitive when developing the intervention through gathering feedback around language and relatable examples. 41,42,53,54,58Researchers described how the inclusion of stakeholders from diverse (ethnic, cultural, age and gender) backgrounds allowed the development of a DMHI that resonated with and reflected the wider target populations.Such alignment of the app with the end users' language, cultural and context enhanced the relevance, acceptability, and usability of the app. 41,53,54richment of ideas.The diversity of stakeholders enriched the variety of ideas and views beyond the researcher's perspective. 49,56,66Indeed, Berry et al. 66 reported that some features included in the Actissist app would not have been considered by the research group without multiple stakeholder input, such as recovery videos.Co-production activities ensured that a range of stakeholder views and needs captured key information and skills that could be integrated into DMHI development, ensuring a more creative and innovate app could be developed. 41,57ceptability and usability.Co-producing DMHIs with stakeholders facilitated the development of useful, relevant, and acceptable apps that met both the end users' and researchers' needs and concerns. 40,45,51,58,60Specifically, Newton and colleagues 51 described how the co-production process resulted in an app that felt relevant and acceptable to both clinicians and service users, showing promise in exposure interventions outside of therapy.Several studies also emphasised the necessity of iterative testing, to identify and incorporate the needs and features identified and prioritised by stakeholders, such as design and content flaws noted in prototypes, in a timely manner. 46,53,64For instance, one study found that after end users engaged in laboratory prototype testing of an initial version, larger visual aids were required for their population of older people. 63dapting the app with larger visual aids improved the appearance and navigation functions of the app, thus increasing its usability and acceptability.The co-production process also afforded the research team insight into which content and design features were/were not acceptable, and generated concrete recommendations for improvement before the app went live, such as simplifying information, 41 modifying terminology 53 and adding goal setting functions. 66herefore, reviewing content with stakeholders early in the design process enabled refinement of content that balanced usability and informativeness, in line with the evidence base.Stakeholders also reported co-production methods increased their trust in the DMHI being developed. 49thods promoting engagement.Certain methodology, including creative methods, such as 'think aloud' interviews were felt, by researchers, to improve engagement and increase honest feedback. 39,53,54Christie et al. 54 changed their methodological approach after noticing a reluctance for end users to express views during interviews and focus groups.However, the employment of 'think aloud' methods, whereby the young person immersed themselves in the app and then gave one-to-one feedback in response to more targeted questions, facilitated engagement, 54 and researchers felt feedback was more authentic and meaningful.Terp et al. 39 also described creative workshops using several design artefact techniques (storyboard, card sorting, mock-ups and paper prototypes) supported active participation, specifically the use of card sorting, where stakeholders arranged 'problem statements' into categories, which supported the position preferences for the content of the app.
A sense of community.A further reported benefit of co-producing smartphone apps was the opportunity to collaborate with stakeholders as part of a community, with mutual engagement where stakeholders' voices felt heard.Stakeholders appreciated the facilitation of connections with others, 43,60 the exchange of ideas 49,52,56 and the enthusiasm it generated. 39,45,47,57For example, end users described feeling seen as knowledgeable resources instead of a patient in need of care. 39

Discussion
The objective of this review was to provide an overview of the methods, and approaches to co-producing mental health app interventions, alongside the barriers and facilitators to co-production.This study represents, to the best of our knowledge, the first systematic review to assimilate the literature and provide a robust summary of the co-production activities of DMHIs via apps.This review highlights the existence of notable gaps of evidence concerning co-produced DMHIs.Specifically, there is significant variability in terms of stakeholder groups involved, stage and role of involvement, methods used, consideration of frameworks, and attempts to minimise power dynamics.The review identified four distinct yet overlapping concepts A key finding was the limited reporting of stakeholder demographics, co-production frameworks, attempts to ensure inclusivity, attempts to minimise power dynamics, and the equality in decision making.This is a common criticism within the research literature where authors rarely detail the activities taken to align with the key principles of co-production. 67Possible reasons for the limited reporting could be due to the absence of a uniformed co-production framework to apply co-production in practice.Alternatively, this may be explained by publication word limits, especially given that most studies' objectives were around the co-produced intervention content and rarely primarily aimed at reporting the co-production process.The findings of this review call for researchers to report co-production activities to allow for co-production principles to be better operationalised in practice.
The value of involving stakeholders throughout all phases is well documented for producing relevant and useful DMHIs for people experiencing mental health difficulties.Most studies (n = 23) involved end users at a 'co-thinker' level, and sometimes as 'partners', but never 'decision makers', suggesting there is a still a need to improve equality between end users and researchers in the co-production process.Phase of involvement was variable, with highest user involvement during the first stage of the co-production process, and only nine studies involving end users throughout all stages of co-production.The findings from this review overlap with the systematic review conducted by Veldmeijer et al., 31 who explored the extent to which stakeholders were involved in mental healthcare through design, including DMHIs.Both reviews found that the level of involvement of end users was lower in the latter phases.This finding aligns with the wider healthcare co-production literature where co-production involvement tends to be limited to the early stage, with less involvement during the middle 'design' phase. 24,68ccording to Sanders et al., 16 active participation is a well-known challenge to successful co-production.A novel finding of this review was that certain methodological approaches such as 'think-aloud' techniques increased active participation and authentic engagement amongst individuals with mental health difficulties.Additionally, creative methods such as the utilisation of storyboarding alleviated the challenge of translating hypothetical ideas into practice.Therefore, more novel methodologies have the potential to improve co-production engagement 69 and may also circumvent power issues and other difficult group dynamics that manifest in focus groups.
A dominate discourse in the wider literature of the challenges limiting co-production activity, are budget costs and time restrictions. 70This review found resources challenges were particularly prominent in the context of digital intervention due to the cost involved in their production.Building an app takes several months and has average related costs ranging from $60,000 to $250,000, 71 with increased prototype developments increasing fees.Thus, adequate budgeting for prototypes, along with realistic time expectations to incorporate feedback, needs to be considered when co-producing apps.Prototype costs is of particular concern given the findings of this review suggest prototype versions of apps that are more closely reflective of the end-product are important to increase user engagement, and to ensure ideas translate into practice.
The included studies highlighted the need to recruit larger, heterogenous samples.A clear concern was the lack of diversity amongst stakeholders, and the co-production sample not being representative of the target audience.This concern is well documented across health research, whereby those involved in research are unrepresentative of the wider audience, especially ethnic minority groups. 72,73Whilst a larger, diverse sample may be more representative and encompass a wide range of ideas and preferences, 74 it requires a process of careful negotiation for researchers to prioritise increased and varying ideas. 72here is significant value of co-production to maximise DMHIs acceptability and engagement through allowing end users voice's to be heard.Based on the good practice of co-production from the studies as evaluated through the bespoke quality tool, recommendations for co-producing DMHIs are made and presented in Table 5. Key highlights include the recruitment of a diverse population involving multiple stakeholder groups, involvement throughout all stages, steps taken to minimise power and promote equality in decision making, employment of think aloud and creative methods to facilitate participation and outlining nonnegotiables of adaptions from the outset (i.e.ideas that interfere with the fidelity of the intervention).
The purpose of this review was to provide an overview of the methods and approaches to DMHIs.Therefore, broad search terms were used to include as many studies as possible that cover the topic and provide a complete overview.The number of included studies, the breadth of co-production methods and user involvement included in this review are a clear strength.A further strength is the development of a co-produced bespoke quality appraisal tool, and • Aim to recruit a diverse population, as cultural adaption of DMHIs has been identified as a factor that increases the effectiveness of DMHIs.• Use multiple stakeholder groups to ensure a diverse range of views are captured (e.g., include parents when developing apps for adolescent populations).

The role and phase of involvement
• Involve stakeholders throughout all stages of the co-production process, from the initial stage of 'discover' where ideas are generated, through to the implementation stage.• Particular consideration should be given to the middle stage of 'develop' to ensure end-user voices are incorporated throughout every stage to develop an acceptable and useable DMHI.

Inclusivity and power
• Proactively take steps to minimise power dynamics.For example, upon the set-up of the co-production group, acknowledge the power dynamic and make attempts to minimise the power differential by, for example, referring to members by names, not professional titles.• Support end-users to lead with decision making.
• Co-production activities should be inclusive and accessible, such as adapting resources and sneering the equipment needed for meaningful involvement is supplied (e.g., tablet access if groups occur online).

Use of creative methods
• Use collaborative 'think aloud' and other creative methods where possible, as traditional focus groups can inhibit participant responses, particularly in younger populations.
5. Process of negotiation • Consider feedback from all stakeholders, but balance this with fidelity to the evidence-based intervention in development.It might not be possible to meet all user group's needs, which could negatively influence co-production experiences.• Outline the non-negotiables of the intervention in terms of theory and rationale at the outset of the co-production process.An open, honest, and sensitive approach should be taken to discuss why any changes were not possible, and where possible, compromises should be made as a collective.• Guide negotiations in a way that manages expectations whilst fostering the relationship with stakeholders and minimising power dynamics.
whilst the tool is not validated, it was able to capture a much richer understanding of the approaches to co-production.Given that the search was limited to studies written or translated in the English language and those published in peer-reviewed journals, publication and language biases are acknowledged.Inclusion of other languages may have yielded more studies focused on low and middle-income countries.Nevertheless, a variety of studies conducted internationally were identified and included.It is important to note that included studies may be influenced by selection bias.Only studies that documented the co-production process were included.Thus, studies that have co-produced DMHIs, but did not report on the process, have not been captured.Therefore, the results need to be considered within this context, and this review may over-represent the co-production that is happening more widely.

Conclusions
Findings suggest there is huge variation in terms of how co-production is being facilitated.Stakeholder involvement and the barriers and facilitators faced when co-producing DMHIs are emphasised.Based on the findings, important considerations for co-producing DMHIs are highlighted.Key recommendations for co-producing DMHIs include involvement of a diverse stakeholder group, involvement at all stages, inclusivity and roles to be considered, the balancing of suggestions and fidelity, and the inclusion of collaborative creative and 'think aloud' methods to allow for authentic feedback.Overall, a range of stakeholders, throughout the entirety of the design, development, and employment of DMHIs through various methodologies is advocated by this review to lead to evidence based and effective DMHIs for people with mental health difficulties.
Considering the insights gleaned from this review and the wider literature, it is imperative that forthcoming research directs its attention towards understanding stakeholder's perspectives of co-production.Additionally, investigation should be undertaken to evaluate the efficacy of implementing a co-produced framework across all four developmental phases of DMHIs.Consequently, empirical testing is warranted to ascertain whether DMHIs developed through co-production yield superior clinical outcome.Additionally, the co-produced bespoke tool used to assess co-production methodological quality should be used as a framework for researchers and app developers to implement co-production in adherence to the core principles.
Contributorship: RU, SB and KB conceived this study.RU researched literature.RU, SB and KB were involved in protocol development.RU was involved in data analysis and wrote the first draft of the manuscript.RU and AB were involved in developing the quality appraisal tool.All authors reviewed and edited the manuscript and approved the final version of the manuscript.

Figure 1 .
Figure 1.PRISMA flow diagram of systematic search.
pertaining to the barriers of successful co-production of DMHIs: (a) resource constraints; (b) recruitment challenges; (c) conflicting views; and (d) power imbalances.The added value of co-producing DMHIs was identified as: (a) enhancing cultural sensitivity; (b) enrichment of ideas; (c) increased acceptance of DMHI; (d) methodology promoting engagement; and (e) a sense of community.

Table 1 .
Search terms by category across all databases.
and abstract.The remaining 107 articles were reviewed for full text screening.Twenty-six papers detailing 24 DMHIs were identified for inclusion in the final synthesis.Included studies were published between 2013 and 2023.

Table 4 .
Methods used for co-producing.

Table 5 .
Clinical implications and recommendations.